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Abstract 

Linear NDCG is used for measuring the performance of the Web content 
quality assessment in ECML/PKDD Discovery Challenge 2010. In this pa- 
per, we will prove that the DCG error equals a new pair-wise loss. 
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1. Linear NDCG 

In ECML Discovery Challenge 2010, the evaluation measure is a variant 
of the NDCG (NDCG 13 ). Given the sorted ranking sequence g and all ratings 
{ r i}\=i, the discount function and NDCG are defined as (r, G {0, 1, . . . , L — 
1}): 

\s\ 

DCG$ = 5>(|S1 - i) , NDCG? = —jDCGt, (1) 
i=i 71 

where DCG^ is the normalization factor that is DCG in the ideal permutation 
7T (DCGP < DCG%). We call ADCG 13 = DCG? - DCG? as the DCG error. 

Specially, DCG^. = mn + m ^~ 1 ^ for the bipartite ranking. It is worth 
noticing that the above NDCG is different from the classical NDCG for the 
query-dependent ranking, where the DCG function is (for the single query): 

DCG ", = | l^T) • NDCG " = l>k= DCG s ■ P) 

Consider the case of the query- dependent ranking with L ratings. For 
the given query, the dataset S can be divided into {Si}^'^ according to the 
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ratings of the instances. Generally, we can define the empirical error for the 
multi-partite case: 

\S a \ \Sb\ 

k w = z E EE^-^^^/K)]- ( 3 ) 

0<a<b<L i=l j=l 

where Z = J2 0<a<b<L \S a \\Sb\. Specially, we also define the following unnor- 
malized empirical error: 

\s b \ 

r w= E EE( i - fl )W < W < 4 ) 

0<a<b<L i=l j=l 

2. NDCG 13 and Pair- wise Loss 

In this section, we will prove the following conclusion: 

ADCG 13 = R(f). (5) 

Theorem 1. For L-partite ranking problem, the unnormalized empirical er- 
ror can be divided into the following form: 

\S a \ \S b \ L-2 
0<a<b<L i=l j=l fc=0 

fc L-l |Sa| | Si, | 

^(/) = E E EE J [/(^) < /(*")]■ ( 7 ) 

a=0 6=fc+l i=l j=l 

Proof. For the convenience of the description, we represent the conclusion 
as follows: 

G\f) = £>(/) 

fc=0 

L-2 fc L-l |S a | | S 6 1 

= EE E EE«<«1 

fc=0 a=0 f>=fc+l i=l j=l 

= R L (f) (8) 
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Now we prove the conclusion G n (f) = R n {f) with the mathematical 
induction on the variable n. If n = 2, the conclusion trivially holds. Assume 
that the equation is true for n, then we will prove the conclusion for n + 1. 
We have 

n-2 k \Sa\ \S b \ 

k=0 a=0 i=l j=l 

n-l \S a \ \S b \ 

a=0 i=l j=l 

n—1 k \S a \ \Sh\ 

= + </(<)] 

k=0 a=0 i=l j=l 

(9) 

and 

n-l \S a \ \S b \ 

R n+1 u) = R n u) + J [/(^ n ) < /(*?)]• ( 10 ) 

a=0 i=l j=l 

Finally, we can prove by the mathematical induction that the second item 
of the right side in fl9]) equals to the corresponding item in (fTOj) . We can see 
that for n = 1 it is trivially hold. 

It follows that G L (f) = R L {f) for all natural number with L > 1. 

Lemma 1. For the bipartite ranking problems, any sorted ranking sequence 
from S = {S + , S-} can be obtained by exchanging at most k = min{|S' + |, |>S_|} 
times from the ideal ranking sequence. 

Proof. Given that there are r(r < m) negative instances in the first m 
positions and s(s < n) positive instances in the remain n positions. 

Now we prove s = r indirectly through the apagoge. If s ^ r, without 
loss of generality, we assume r > s. It is known that there are r — s negative 
instances in the first m positions after s exchanges. The exchanges occur 
among s negative instances in the first m positions and s positive instances 
in the remain n positions. Then the fact that we will get r — s + n negative 
instances is in contradiction to n negative instances. Finally, we can conclude 
that r = s < min{\S + \, l^-l}. 

Next, we will prove 
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Theorem 2. For the bipartite ranking problem, DCG errors with [I] equals 
the unnormalized expected losss R(f): 

m n 

ADCG? = R(f) = /[/(*,+) < f{x-)]. (11) 

i=l j=l 

Proof. We know that any ranking sequence can be obtained by the ex- 
change operations from the ideal ranking sequence according to Prop. [TJ Let 
{*i>*2, •••>«*}(! < k < k < ■ ■ ■ < ik < m) and {j u j 2 , ■ • • ,jfc}(l < ji < 
j 2 < . . . < j k < n) be the exchanged positions in the first m positions and 
the remain n positions, respectively. As depicted in Fig. (TJ without loss of 
generality, we exchange i r and j r for the r-th time. First, we will compute 
the decrement relative to the ideal ranking sequence for the r-th time 

A r DCG = (m + n — i r ) — (m + n — (m + j r )) 

= m + j r — i r >=l. (12) 

Now, we give a detailed explanation about the increment of the unnormalize 
expected loss which is related to the position i r and j r . The increment due 
to the variation in the position i r will be m — i r + r because there are m — i r 
positive instances in the first m positions and r positive instances in the 
remain n instances. As for the position j r , the increment should be j r — r 
since there are j r — 1 — (r — 1) negative instances in the remain n instances 
before j r . In summary, we obtain the increment A r R(f) = m + j r — i r . As 
a result, we conclude that 

k k 

ADCG 13 = &vDCG = A rR{f) = AR{f). (13) 

r=l r=l 

Notice that the initial value of R(f) (the ideal ranking sequence) is zero, this 
proves the theorem. 

Fig. [2] gives an example to verify the conclusion ADCG 15 = R(f) = 4. 
The following theorem shows that the conclusion ADCG 13 = R(f) still holds 
when extending to the multi-partite ranking problem. 

Theorem 3. For L-partite ranking problem, the DCG errors with Eqn. (QJ) 
equals R(f): 

\Sa\ \S b \ 

adcg?= EE^-^i/^^W)- ( 14 ) 

0<a<6<L i=l 3=1 
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Figure 1: The ideal ranking sequence with its transformation. Left: the ideal ranking 
sequence, right: the ranking sequence with multiple exchanges 

7T + + + - - - 

il 2 3 4 5 6 
9 + - - + + - 

Figure 2: The example on the bipartite ranking shows ADCG@ = R(.f) = 4, where 
DOG.* = 12 and DCG = 8. 



PROOF. From[U we know that 



L-2 



R(f) = G(f) = Y,Rk(f)- (15) 

fc=0 

Then we will show that DCG in L-partite problem can be written as the 
sum of the DCG measures of L — 1 bipartite problems. We divide DCGp 
into 

DCG/3 = 5> t (|S|- ? ) 

i=l 

\S\ L-2 

i=l k=0 
L-2 



J2J2 I ( k<r ^ s \ 

i=l k=0 
L-2 

Y,DCG k , (16) 



fc=0 



where DCG k = jjfJi ![ k 

< ri\(\S\ — i). For given k, we can assign the 
instances with rj (k < rj) to the ranking 1 and the others to the ranking to 
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obtain a bipartite ranking problem with the unnormalized empirical error 

k L-l \S a \ \S b \ 

Mf) = £ E EE J (/( x ') < /(**)]■ 

a=0 fe=fc+l i=l j=l 

From El ADCG k = R k (f) holds. We have ADCG = T,t=o ADCG k = 

EtZ 2 Rk(f) = R(f)- 



it 2 2 1 
i 1 2 3 4 5 6 
5 2 2 1 

Figure 3: The example on the multipartite ranking shows ADCGp = R(f) = 3, where 
BCG'i = 21 and DCG^ = 18. 

The example in |3] supports our conclusion about the DCG error and the 
unnormalized expected loss in the multipartite ranking problem. 



6 



